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ASSESSMENT  Of  SUBJECTIVE  ?BD BASIL ITT* 

Richard  E.  Barlow 

Operation*  Research  Cantor 
University  o f  California 
Berkeley,  California  94720 


Abacract 

Tha  uitflMQC  of  subjective  probability  la 
of  great  lntaraat  in  risk  analysis.  Soma  alda  for 
aaaaaaing  subjective  probability,  ara  survey ad . 

Tha  connection  with  statistical  Inference  and  re¬ 
cent  papers  on  statistical  foundations  ara  dis¬ 
cussed. 

I.  Introduction 

Recant  DOS  and  NRC  sponsored  projects  have 
involved  tha  assessment  of  subjective  probability* 
For  example,  tha  Seismic  Safety  Margins  Research 
Program  at  Lawrence  Livermore  National  Laboratory 

concerning  nuclear  power  plant  safety  has  Involved 
assessing  expert  opinion  relative  to  the  strength 
of  critical  components  (George  and  Manning1) . 
Combining  expert  opinion  was  necessary  due  to  the 
lack  of  an  experimental  data  base. 

EP1I,  Bechtel  and  others  have  been  Interested 
In  coal  fired  electrical  power  plant  availability. 
Data  on  rare  failure  events  are  again  difficult 
to  obtain.  Especially  In  designing  a  new  power 
plant,  subjective  failure  probability  assessments 
are  required. 

A  third  area  of  experimental  research  where 
subjective  assessments  are  required  la  that  of 
extrapolation  procedures  for  late  effects  of  radi¬ 
ation  at  low  doses  (P.  G.  Groer  and  R.  E.  Barlow2). 
The  human  health  effects  of  low  doses  of  Ionising 
radiation  have  been  the  subject  of  much  contro¬ 
versy.  A  recognition  of  tha  inevitable  subjec¬ 
tive  assessments  Involved  might  have  mitigated 
of  this  controversy. 


Problems  of  the  above  kind  ere  not  about  to 
go  away,  tnacaad,  they  will  in  all  It kali hood 


Work  supported  by  the  Air  force  Office  of  Scien¬ 
tific  Research  under  Grant  AF0SR-S1-0122  and  the 
Me chems tics  sod  Statistics  Division,  Lswrsncs 
Livernors  Rational  Laboratory. 


bacons  even  more  important  end  pressing  In  future 

years. 

Bow  ere  these  probability  assessments  to  bs 
mads?  It  la  sasy  to  criticise  the  current  litera¬ 
ture  concerning  probability  assessment  methodology 
end  I  disagree  with  many  of  the  methods  used. 
However,  some  groups  of  people  have  been  assess¬ 
ing  probability  for  e  long  time.  Examples  are 

•  Odds  Makers:  H Jimmie  the  Greek" 

•  Weather  Forecasters 

•  Actuerlss 

•  Ssismologists. 

All  of  us  respect  the  odds  assessments  of  “Jlnle 
tha  Grsek."  Wa  may  have  lees  respect  for  weather 
forecasters.  A  school  of  mathematical  actuaries 
la  concerned  with  so-called  credibility  formulae. 
These  are  formulae  for  determining  insurance 

premiums  baaed  on  experience  as  well  ae  subjective 
3  4 

judgement,  e.g.  Biihlmann,  Jewell.  This  school, 
however,  seams  divided  between  subjective  Bayeslaas 
end  empirical  Bayaalana.  A  prominent  seismologist 
has  recently  assessed  tha  probability  of  a  great 
earthquake  in  the  state  of  California  within  tha 
next  tea  years  to  be  one-half.  In  many  cases, 
though  perhaps  not  in  this  instance,  a  probability 
of  one-half  la  used  when  the  forecaster  has  little 
knowledge  concerning  the  event  in  question. 

II.  Aseessment  Aida 

Bruno  da  Flaattl  was  one  of  the  earliest 
exponents  of  subjective  probeblllty.  Hie  ideas 
(cf.  de  Finetti,5  Chapter  5)  ere  the  beele  for 
much  of  the  current  research  on  probability  assess¬ 
ment  alda.  Some  of  tha  most  promising  are: 

1.  linear  Programing  as  an  eld  la  checking 
probeblliatlc  consistency  as  well  ae  for 
p lor  lag  the  probeblliatlc  consequences  o 
assessed  probabilities; 
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2.  Scoring  Rule*  mm  aid*  in  encouraging  serious 
probability  assessment  as  trail  as  evaluation 
of  assessments  in  tamo  of  actual  avaat  out- 

coaas; 

3.  Methods  for  Combining  Expert  Opinion .  Lindley 

6  7  8 

at  al. ,  Lindley  and  Lindley  ara  some  of  tha 
base  rafarancaa  coneamad  vith  combining  ax* 
part  opinion. 

III.  Linaar  Programming 

For  tha  laat  two  yaarav  I  hava  asked  ay  re- 
llabillty  class  to  assass  thair  probabilities  of 
tha  U.C.  Basra  winning  certain  football  gaskaa. 

For  example,  I  aakad  thair  probabilities  for  tha 
(conditional)  event  that  the  Bears  beat 

Washington  Arizona  |  UCLA  |  •  Ej 

*x  *2  '  *2  *3  '  h 

Pi  *2  P3 

where  p^  -  PCE^  |  F^]  and  F^  is  tha  aura 
event.  •  1  if  tha  Bears  beat  Washington  and 
1^-0  otherwise.  EjF^  *  1  if  tha  Bears  beat 
Arizona  and  Washington,  so  that  p2  is  tha  con¬ 
ditional  probability  they  beat  Arizona  given  they 
beat  Washington.  Of  course  tha  Bears  usually  lose 
and  we  quickly  lose  interest.  However,  before  tha 
gaaas  ara  played,  tha  students  construct  tha  fol¬ 
lowing  table  of  possible  outcomes: 


(1.1)  (1.1)  (1.1) 

signifying  tha  Bears  won  all  these  gaaas  and  tha 

conditioning  events  of  course  ware  also  true  in 

this  case.  Tha  students  than  use  a  linaar  prog- 
o 

ran  devised  by  Bob  Kau  to  determine  whether  or 
not  thair  assessed  probabllitls ;  ara  consistent. 
If  there  exist  w^  >  0  ,  J  •  1,2,  ....  8  such 
that 

^mih  Wj/X  Vi  a: 

then  the  student's  probabilities  ara  consistent 

(we  say  ha  is  strictly  coherent) .  If  w.  >  0  , 

*  J  " 

and  soma  w^  -  0  ,  than  tha  student  Is  only  co¬ 
herent  and  ha  has  Implicitly  assigned  probability 
zero  to  a  logically  possible  outcome.  If  no  such 
w^'s  exist,  then  he  la  inconsistent  (we  say  he 
is  incoherent)  and  should  revise  his  opinion. 

9 

In  his  Fh.D.  thesis.  Bob  Hsu  formulated  the 
problem  of  determining  (strict)  coherence  as  a 
linear  programing  problem.  This  was  done  in  the 
context  of  a  two  person  zero  sum  gam  involving 
a  Bookie  (the  probability  assessor)  and  a  Bettor 
(his  adversary).  The  Bookie  sacs  fair  prices  p^ 
1  -  1,2,  ...,  n  for  one  dollar  payoffs  on  event 
pairs 

(^l'^i)  '  ••••  • 


Table  I 


Implicit  \ 
Probabilities  \ 

»i 

<W 

p2 

*3 

(e3.f3> 

"i 

®U,,U> 

(E21‘F21> 

(e31.f31) 

w2 

^*12**12* 

(*21,?22^ 

(*32,f32) 

»1 

«2S*V 

^sa'V 

Tha  J6*1  possible  outcome  corresponds  to  the  pairs: 

<*irV  •  <W  •  <VV  ■  \rl  a 

also  occurs  la  the  J6*  possible  outcome  and 
Sjj  •  0  otherwise*  For  example,  the  first  row  of 
the  table  could  be 


The  bettor  is  allowed  to  make  z^  bets  on  (or 
against)  the  event  pairs  *  the  Bookie 

charges  the  Bettor  p^z^  dollars  for  ehe  privi¬ 
lege.  If  EjF^  •  1  ,  the  Bookie  pays  z^  dollars 
to  the  Bettor.  If  F^  -  0  ,  the  Bookie  refunds 
che  Bettor  his  money  and  the  bet  is  called  off. 

If  •  0  and  F^  •  1  ,  the  Bettor  receives 
nothing.  The  Bookie's  oat  gala  is  then 


I 


t-i 


■Wi 


which  eu  of  coutm  b«  oag.clv..  If  ch.  Jth  ouc- 

come  occurs,  let  the  Bettor  receive  y  +-  y.  and 

®  3 

hence 


>.*>,*£  vwi-° 


since  it  is  a  sere 


3- 


Given  ....  pn  ,  consider  Che  problem 

Maxi  ml.  aa  y^  q 

subject  to  yo  +  r3  *  l  (Pt  *Etj)liFij  "  0 

a  1  *  1*2,  * • • *  a  » 

l  Mi  '  1 

j-i  :  3 

i  0  j  •  1*2*  ....  *  . 

Ve  muse  also  specify  Bj  >  0  ,  j  •  1,2,  •  •»,  a 
(the  Bettor's  probabilities  on  outcome).  Usu¬ 
ally,  ve  choose  fl,  -  B,  -  ...  •  §  •  1/a  .  Let 

*  12a 

yg  be  the  solution  to  our  problea.  Then  the 

Bettor  is  guaranteed  to  make  at  least  y„  .  It 
*  0 

turns  out  chat  if  y  >  0  ,  ehe  Bookie  la  tacon- 
sis cent  and  he  should  revise  his  prices  ("prob¬ 
abilities")  .  If  y*  •  0  ,  the  Bookie  is  coherent 
and  if  y^  <  0  he  la  strictly  coherent. 

By  using  this  linear  programing  approach, 
the  probability  assessor  can  quickly  determine 
if  his  probabilities  are  consistent.  He  can  also 
explore  the  consequences  of  his  probability 
assessaants  by  performing  a  sensitivity  analysis 
vich  the  linear  program  and  its  dual. 

IV.  Scoring  Rules 


Hot#  can  the  probability  assessor  be  en¬ 
couraged  to  think  carefully  about  his  probability 
assessments?  One  answer  may  be  the  scoring  rule. 
For  example,  suppose  a  weather  forecaster  states 
his  probabilities  ,  r2  ,  for  the  events 

Rain  Snow  Sun 


Suppose  further  chat  the  events  are  carefully 
defined  so  that  they  are  mutually  exclusive  and 
exhaustive.  The  so-called  Brier  scoring  rule  is 


To  obtain  a  rule  reflecting  the  user's  (say 
a  farmer’s)  concerns,  let  u^  be  the  user's 
utility  if  he  makes  decision  d  and  outcome  k 
(say  snow)  occurs.  Given  forecast  r  -  (r^r^r^)* 
the  user  may  then  calculate 


C(r> 


k-1 


“dkrk 


to  determine  his  best  decision.  Vote  that  G(r) 
la  convex  in  r  but  not  strictly  convex.  Let 

G*(r)  -  (G^(r),  ....  G*(r)) 

be  a  subgradient  to  G  at  r  ;  i.a. ,  for  all 
forecasts  p 

G(p)  >.G(r)  +<p-r  ,  G*(r)  > 
where  <  >  denotes  inner  product. 

How  suppose  outcome  k  occurs  and  p  la  the 
perfect  forecast  for  k  ;  i.a.,  p^  •  1  while 
Pt  •  0  for  i  +  k  .  Let 

-  G(p)  -  G(r)  -  (p  -r  ,  G*(r) ) 
be  our  scoring  rule.  Then  in  this  particular  case 


G(p) 


dk 


and 


Vr> "  “dk  ‘  “ik  (3> 

d 

n  n 

wh«»  l  u..  r.  i  l  u  .  r.  for  tU  J  *  l  . 
k-1  k-1 

Clearly,  (3)  is  the  value  to  the  user  of  perfect 
information  minus  the  value  of  information  con¬ 
tained  In  r  when  k  occurs.  This  scoring  rule 
has  a  property  which  Savage^  called  proper.  Ve 
•ay  that  any  •coring  rule  is  (strictly) 

proper  iff 


I  Vk(r)  L  E  rkVq) 

k.1  *  *  («)  k.1  *  * 


I  "i  v  o: 

where  Z^  •  1  if  E^  occurs  and  •  0  other¬ 
wise.  daarly,  aeoraa  doaa  to  sere  ara  beat. 
This  rule  has  been  advocated  aa  a  aathod  for 
scoring  weather  for  teas  ears. 


for  all  foracaata  r  and  q  •  Henca,  if  you  be- 

liava  r  but  hadga  and  say  q  ,  you  will  txpact 

to  receive  a  worse  score  than  if  you  are  honest 

and  say  r  .  Scoring  rule  (3)  is  not  strictly 

proper.  However,  (2)  is  strictly  proper.  L.  J. 

10 

Savage  explored  In  depth  the  use  and  properties 
of  scoring  rules  aa  aide  la  probability  assessment. 
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H*  (somewhat  loosely)  characterized  proper  end 
strictly  proper  scoring  rules.  Eduerdo  Haim11  In 
his  Ph.D.  thesis  clarified  and  extended  Savage's 
work  in  several  directions. 

V.  The  Bayesian  Paradigm 

I  spent  the  1975-76  academic  year  at  Florida 
State  University  In  Tallahassee.  My  purpose  was 
to  complete  a  book  on  Statistical  Reliability 
Theory  with  Frank  Proschan.  At  the  time.  X  was 
working  on  total  time  on  test  processes.  I  saw 
this  as  a  way  of  unifying  life  test  procedures. 

At  the  same  time,  I  started  attending  lectures  by 
Dev  Besu  on  statistical  Inference.  It  was 
Lehmann's  hypothesis  testing  course  and  Lehmann's 
book  was  the  text.  However.  I  noticed  something 
strange  -  Basu  nrver  opened  the  book.  He  was 
obviously  not  following  It.  Instead,  he  was  giv¬ 
ing  a  very  elegant,  measure  theoretic  treatment 
of  the  concepts  of  sufficiency,  anclllarlty,  and 
Invariance.  He  was  interested  in  'he  concept  of 
information  -  what  It  meant  -  how  it  fitted  In 
with  contemporary  statistics.  As  he  looked  at 
the  fundamental  Ideas,  the  logic  behind  their  use 
seemed  to  evaporate.  I  was  shocked.  I  didn't 
like  priors.  I  didn't  like  Bayesian  statistics. 
But  after  the  smoke  had  cleared,  that  was  all 
that  was  left. 

12 

Berger  at  Purdue  University  has  recently 
written  a  new  graduate  level  test  on  decision 
theory.  In  his  preface,  he  says  that  he  had  in¬ 
tended  to  adopt  a  neutral  position  vls-a-vls  the 
various  statistical  approaches.  However,  In  the 
course  of  writing  the  book,  he  turned  Into  a 
rabid  Bayesian.  He  says  "There  was  no  single 
cause  for  this  conversion;  just  a  gradual  reali¬ 
zation  chat  things  seemed  to  ultimately  make 
sense  only  when  looked  at  from  the  Bayesian  view¬ 
point." 

Basu  loves  counterexamples.  He  Is  like  an 
art  critic  In  the  field  of  statistical  Inference. 
He  would  find  a  counterexample  to  the  Bayesian 
approach  If  he  could.  So  far,  he  has  failed  in 
this  respect. 


Recently,  Basu1^  wrote  the  following:  "It 
Is  about  12  years  now  that  I  finally  came  to  the 
sad  conclusion  that  most  of  the  statistical  meth¬ 
ods  that  I  had  learned  from  pioneers  like  Karl 
Pearson,  Ronald  Fisher  and  Jerzy  Neyman  and  sur¬ 
vey  practitioners  like  Morris  Hanson,  P.  C. 
Mahalanobis  and  Frank  Yates  are  logically  un- 
tenable."  I  believe  he  is  right.  Read  Baeu.^ 

At  the  beginning  of  this  talk,  I  was  con¬ 
cerned  with  the  problem  of  assessing  probabilities 
for  seemingly  very  specialized  problems  of  current 
Interest.  However,  from  the  Bayesian  point  of 
view,  this  seems  to  be  pretty  much  the  problem  of 
modern  statistics.  After  modeling,  the  problem  of 
statistical  inference  is  the  assessment  and  cal¬ 
culation  of  probabilities.  However,  this  is  not 
a  trivial  problem! 

As  I  understand  It,  some  of  the  main  points 
of  the  Bayesian  paradigm  are: 

1.  All  probabilities  are  subjective.  However, 
the  calculus  of  probability  is  essentially 
the  same  as  for  frequenclsts; 

2.  All  probabilities  are  conditional  on  current 
information  available  to  the  analyst; 

3.  Probability  is  a  measure  of  the  analyst's 
uncertainty  about  unknown  quantities; 

4.  Information  is  anything  which  changes  the 
analyst's  probability  distributions  about 
unknown  quantities  of  Interest. 
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